In these days, the compression and coding technology for image data has progressed remarkably. The compression and coding technology is effective not only in an efficient use of a storage medium, but also in shortening the time for transmitting the image data through network. When original image is encoded at high compression ratio, it is general that the irreversible image compression method is adopted even though the reconstructed image is not accord perfectly with the original. It is designed in most of the irreversible image compression methods that an image data is divided into a plurality of blocks per M×N pixels unit, the orthogonal transform is performed in each block, the obtained orthogonal transform coefficient is quantized and then the image data is encoded. The typical irreversible image compression method is JPEG, which is widely in use as a compression method for color statistic images.
The outline of the encoding and decoding processing of JPEG compression is explained according to FIG. 1. In case of JPEG compression, a color transforming unit 10 transforms each pixel data consisting of multi-valued data (density data) of Red(R), Green(G), and Blue(B) to data of luminance component (Y) and color difference (Cr, Cb). There are many definitions of RGB space. But in case of sRGB space, one of the RGB space definitions, the transform from RGB data to YCrCb data is performed according to a following expression (Expression 1).Y=0.29900×R+0.58700×G+0.11400×B Cr=0.50000×R−0.41869×G−0.08131×B+128CB=−0.16874×R−0.33126×G+0.50000×B+128  Expression 1
Next, a DCT unit 11 performs the Discrete Cosine Transform (DCT) of YCrCb data per 8×8 pixels block unit. Where a DCT coefficient after the transform is defined as DCT[v][u] and YCrCb data before the transform is defined as F[r][x], the DCT processing is performed according to a following expression (Expression 2). Besides, the above “x” indicates a position in the horizontal direction of the original image before the DCT processing in each block, the above “y” indicates a position in the vertical direction of the original image before the DCT processing in each block. And the above “u” indicates a position in the horizontal direction of the DCT coefficient after the DCT processing in each block, and the above “v” indicates a position in the vertical direction of the DCT coefficient after the DCT processing in each block.DCT[v][u]=¼×Cu·Cv·ΣΣF[y][x]·cos((2x+1)uπ/16)·cos((2y+1)vπ/16) Cu, Cv=1/√{square root over (2)}(u,v=0), 1(otherwise)  Expression 2
Besides, since the sensitivities of human eyes to the color difference component is lower than that to the luminance component, the compression ratio increases by sub-sampling the color difference component. In general, the sub-sampling is rendered to correspond one pixel of the color difference component to the luminance component for 2×2 pixels. Accordingly, after thinning out 8×8 pixels data from a block of 16×16 pixels unit, the DCT processing is performed.
Next, the quantizing unit 12 quantizes the DCT coefficient. Where the quantized DCT coefficient is defined as QDCT[v][u] and a value for quantizing each component of DCT coefficient is defined as Qtable[v][u], the quantization is performed according to a following expression (Expression 3).QDCT[v][u]=INT{DCT[v][u]/Qtable[v][u]+0.5}  Expression 3
But, a value of INT{a} is a maximum integer value not over a value a.
Besides, each value in a quantization table 13 can be adopted as a quantizing unit value used in the quantization of DCT coefficient corresponding to each frequency, and said value can be decided by the user's discretion. Generally, the sensitivities of human eyes to the high frequency component is lower than that to the low frequency component, therefore, the quantizing unit value to the high frequency component is relatively larger than that to the low frequency component. And the sensitivities of human eyes also to the color difference component are lower than that to the luminance component. Therefore, the quantizing unit value to the color difference component is relatively larger than that to the luminance component. For reference, the quantization tables recommended by the standard JPEG method are shown in FIG. 2 and FIG. 3. FIG. 2 shows the quantization table of the luminance component (Y), and FIG. 3 shows the quantization of the color difference component (Cr, Cb). Besides, since each quantizing unit value of the quantization table used by the quantization processing is required at the decoding, said values are stored in the encoded JPEG compressed data.
The quantized DCT coefficient is encoded by an entropy coding unit 14. Under the JPEG standard method, Huffman coding is used as the entropy coding.
The above-mentioned steps schematically explain about the encoding from the image data to the JPEG-compressed data. On the contrary, when the JPEG-compressed data is decoded to an image data, the process takes the inverse order to the above encoding steps. The following describes the decoding steps.
First, an entropy decoding unit 15 performs the entropy decoding of the JPEG-compressed data.
And, an inverse quantizing unit 16 performs the inverse quantization. At this time, the inverse quantizing unit 16 reads from the JPEG-compressed data the quantization table 13 used at the encoding, and respective quantizing unit values of the quantization table 13 are used as a value for inverse-quantizing each encoded component. That is to say, referring to an inverse quantization table 17 wherein inverse-quantizing unit values are the same as the quantizing unit value of the quantization table 13 used at the encoding, the inverse quantizing unit 16 performs the inverse-quantization of each encoded component using each inverse-quantizing unit value in the inverse quantization table 17. Where the inverse-quantized DCT coefficient is defined as RDCT[v][u], the inverse-quantization calculation is performed according to a following expression (Expression 4).RDCT[v][u]=QDCT[v][u]×Qtable[v][u]  Expression 4
Besides, since the inverse-quantized DCT coefficient according to the Expression 4 is calculated using the coefficient rounded up by the quantization of the encoding, it does not exactly reproduce the DCT coefficient obtained from the original image. However, the DCT coefficient obtained from the original image is ensured to be over a lower limit value dDCT[v][u] found by a following expression (Expression 5), and under a upper limit value pDCT[v][u] found by a following expression (Expression 6).dDCT[v][u]=(QDCT[v][u]−0.5)×Qtable[v][u]  Expression 5pDCT[v][u]=(QDCT[v][u]+0.5)×Qtable[v][u]  Expression 6
After the inverse-quantization is performed as described above, an inverse-DCT unit 18 performs the inverse-DCT. The inverse-DCT transforms the DCT coefficient to the YCrCb data. Where the YCrCb data after the inverse-DCT is defined as G[y][x], the inverse-DCT is performed according to a following expression (Expression 7).G[y][x]=¼ΣΣCu·Cv·RDCT[v][u]·cos((2x+1)uπ/16)·cos(2y+1)vπ/16) Cu, Cv=1/√{square root over (2)}(u,v=0), 1(otherwise)  Expression 7
In the final step, a color transforming unit 19 performs the color transform from YCrCb data to RGB data, in result a reconstructed image can be obtained. A following expression (Expression 8) is a transform expression used at the transforming from YCrCb data to sRGB data.R=Y+1.40200×(Cr−128)G=Y−0.71414×(Cr−128)−0.34414×(Cb−128)B=Y+1.77200×(Cb−128)  Expression 8
The above steps are the outline of the encoding and decoding of the JPEG compression, but the quantizing step is included in the encoding step, and this degrades the data. The data degradation appears in the reconstructed image as noises, and the picture quality of the reconstructed image becomes worse than that of the original image. FIG. 4 shows an example of the original image, and FIG. 5 shows an example of the reconstructed image decoded from the JPEG-compressed data of the original image.
One of noises seen in the reconstructed image is a mosquito noise shown in FIG. 5 that has a bad visual influence. The mosquito noise is the gradation fluctuation; it seems that mosquitoes are flying around edges of the reconstructed image. Most of the high frequency components drop out due to the quantization of DCT coefficient, so that strong edges existed in the original image cannot be reconstructed exactly, with the result that the gradation fluctuation appears.
And another noise is a block distortion shown in FIG. 5. The block distortion is a phenomenon that the block boundary of the reconstructed image has the discontinuous gradation due to the encoding per block of 8×8 pixels. Such noise appears remarkably in areas of which gradation value changed gradually in the original image.
One of the prior art of removing these noises is the filtering (the smoothing), and by performing the filtering of a reconstructed image the obstructive mosquito noise and block distortion appearing in the reconstructed image are removed. Where the reconstructed image is uniformly subjected to the smoothing by means of such method, the noise can be reduced. However, true edges in the original image get dull, and the reconstructed image becomes blurred.
The method to avoid the unexpected blurred image is disclosed in Japanese Patent No. 2,962,815, namely, at the time of encoding the original image, the compressed data is added with information for specifying a block of which boundary is at the same position of true edge of the original image(edge block). And at the decoding, the blocks in the reconstructed image are divided to the edge blocks and the others. The other blocks are processed so as to remove the bock distortion therefrom, and the edge blocks are processed so as to remain the edge.
However, in the above prior art disclosed in the Japanese Patent No. 2,962,815, when the original image is encoded, the information specifying the edge block must be added to the compressed data. Therefore, the encoding apparatus should be provided with both a function for specifying the edge block and a function for adding said identifying information to the compressed data.
Another method is also suggested besides the above prior art, in which the above mentioned particular functions are not added to the encoding apparatus but functions are added to the decoding apparatus, and said method can restrict to degradation of the true edge of the original image. That is to say, the block distortion and the mosquito noise are removed with controlling the occurrence of the blurring in the image. A particular one of the effective methods is the Projection Onto Convex Set method (POCS method). The POCS method is the iteration of the smoothing and the restricted condition-based projection method by turns. Here is described the processing of the POCS method according to FIG. 6.
After a DCT unit 20 performs the DCT processing of the reconstructed image decoded from the JPEG compressed data, a restriction condition calculating unit 21 calculates restriction conditions for the projection processing. The restriction condition is to restrict respective DCT coefficients of blocks composing a final outputted image within the range of DCT coefficients that the original image might have involved.
In the steps from the JPEG encoding to the decoding, the DCT coefficient is quantized. And the DCT coefficient before DCT processing is assured to be the lower limit value dDCT[v][u], and under the upper limit value pDCT[v][u]. Therefore, as the restriction condition for the projection processing, the restriction condition calculating unit 21 calculates the lower limit value dDCT[v][u] and the upper limit value pDCT[v][u], the deference of the two values indicating the variable range of DCT coefficients (see Expressions 5 and 6).
The following explains a processing for removing the noise from the reconstructed image by the iteration of the projection processing and the smoothing by returns using the restriction condition calculated by the restriction condition calculating unit 21 (the lower limit value dDCT[v][u] and the upper limit value pDCT[v][u]).
A smoothing unit 22 smoothes the reconstructed image uniformly by the filtering. The smoothed image data is subjected to the color transform to YCrCb data by the color transforming unit 23, and then to the DCT processing by the DCT unit 24.
The obtained DCT coefficient is subjected to the projection processing by a projecting unit 25 according to the lower limit value dDCT[v][u] and the upper limit value pDCT[v][u] calculated by the restriction condition calculating unit 21. That is to say, where a DCT coefficient calculated by the DCT unit 24 is less than the lower limit value dDCT[v][u] or more than the upper limit value pDCT[v][u], the DCT coefficient is rounded up to the variable range limit value. That is to say, where the DCT coefficient is less than the lower limit value dDCT[v][u], the projecting unit 25 replaces the DCT coefficient with the lower limit value dDCT[v][u]. And where the DCT coefficient is over the upper limit value pDCT[v][u], the projecting unit 25 replaces the DCT coefficient with the upper limit value pDCT[v][u].
After the rounded DCT coefficient is subjected to the inverse-DCT processing by an inverse-DCT unit 26, a color transforming unit 27 performs the color transforming from YCrCb data to RGB data.
An end deciding unit 28 decides whether the noise removal processing is terminated or not. If it is decided that the processing is continued, the respective composing units from the smoothing unit 22 to the color transforming unit 27 performs the same processing again.
By much iteration of the smoothing and the projection processing based on the restriction condition, the block distortion and the mosquito noise can be removed satisfactorily, however, the true edge of the original image gets dull and the blurring of the image gets worse.
Therefore, the number of times the above mentioned processing iteration, which enables to remove the block distortion and mosquito noise with controlling the dullness of the true edge in the original image, is set in advance to the end deciding unit 28. The end deciding unit 28 decides the termination after iterating the smoothing and the projection processing the predetermined number of times. In result, since the POCS processing controls the dullness of the true edge, it is possible to removes the block distortion and mosquito noise in the reconstructed image.
An image shown in FIG. 7 is an image obtained by processing the reconstructed image in FIG. 5 using the conventional POCS method. FIG. 7 shows the fact that the iteration of the smoothing and the projection processing based on the restriction conditions enables to reduce the noise in the reconstructed image without apparent dull.
Besides, instead of the predetermined number of times, the end deciding unit 28 may decide the termination of the above iteration processing according to a value obtained by the evaluation of the image. For instance, the end deciding unit 28 may decide the termination of the iteration processing when the changing value of the image becomes small after the above iteration processing. To be concrete, where an image processed k times is defined as fk(x, y) and an image processed k+1 times as fk+1(x, y), the changing value E of the image after k+1 times processing can be found by a following expression (Expression 9
                    E        =                              ∑                          x              ,              y                                ⁢                                          ⁢                                    {                                                                    f                                          k                      +                      1                                                        ⁡                                      (                                          x                      ,                      y                                        )                                                  -                                                      f                    k                                    ⁡                                      (                                          x                      ,                      y                                        )                                                              }                        2                                              Expression        ⁢                                  ⁢        9            
When it found this changing value E is under a specific value, the end deciding unit 28 decides the termination of the above iteration processing.
However, in the noise removal processing of the POCS method, the whole image is smoothed uniformly by the same filter; therefore a blurring in some degree appears in the reconstructed image. That is to say, as shown in FIG. 7, the true edge in the original image becomes dull.
In order to settle this problem, the next prior art is designed that the reconstructed image is divided into some areas in consideration with the edges, and the respective areas are subjected to the smoothing using different filters.
Japanese Patent Publication No. 7-170518 discloses that the image is divided into a plurality of small areas by combining neighboring pixels if the difference between the neighboring pixels is less than a specific value (such small area is different from a block, and not always in the same size), and the smoothing is performed per area. According to such method, since the areas are divided at the edge, the smoothing is not performed traversing the edge. Therefore it is possible to obtain an effect that the blurring of the edge is made relatively small.
However, in the above method, there is a possibility that the image is divided to areas also based on the discontinuity of gradation generated as the block distortion on the block boundary. In such case, the block distortion cannot be reduced and removed.
Therefore, this invention considers the above conventional problems and has an object to provide an image processing apparatus enables to remove the block distortion with controlling the blurring on the edge.
The block distortion removal processing disclosed in Japanese Patent No. 2,962,815 is explained here according to FIG. 8.
In the disclosed block distortion removal processing, according to the block specifying information added to the compressed data, blocks to be removed block distortion therefrom are specified from the reconstructed image. In the following explanation, a target block x shown in FIG. 8 is specified as a block to which the block distortion removing is executed.
And as shown in FIG. 8, a corner a of the target block x consists of pixel a1, a2, a3 and a4, said pixel a1 in the target block, said respective pixels a2, a3, and a4 in blocks L, LU, and U neighboring to the target block x. The value of pixel a1 is presumed by averaging all the values of pixels a1, a2, a3, and a4. And a value calculated by subtracting the value of pixel a1 of the reconstructed image from the presumed value of pixel a1 is considered as a correction value of the pixel a1.
Likewise, regarding the other three corners b, c and d of the target block x, correction values for respective pixels b1, c1, and d1 are calculated.
Then, each correction value of pixels in the target block x are calculated by weighting and averaging the correction values of pixels a1, b1, c1 and d1 according to the respective distances from the pixels a1, b1, c1 and d1. And the respective calculated correction values of pixels are added to the values of the corresponding pixels; thereby a block distortion reduced image can be reconstructed.
According to such method, it is possible to obtain the reconstructed image wherein the block distortion is reduced. However, it result in that the value of corner pixel in the target block x and the other values of corner pixels of neighboring blocks are the same one. That is to say, regarding the corner a, the value of pixel a1 of the target block x and the other values of pixels a2, a3, and a4 of the blocks L, LU, U are the same one. Hereupon, the block distortion can be reduced, but the pixels surrounding a corner of block have the same one value, in result the smoothing gradation change over each corner of the block could not obtain.
Therefore, considering the above problems, the present invention also has an object to an image processing apparatus enables to reconstruct the smoothing gradation change over corners of blocks composing the reconstructed image and remove the block distortion.
Japanese Patent Publication No. 8-2184309 discloses a method of filtering the reconstructed image by changing the smoothing filter according to the compression ratio of the compressed data to the original image. It is sure that visually bad influences based on the noises like the mosquito noise and the block distortion depends on the compression ratio of the compressed data to the original image, and also depends on the output size of the reconstructed image. That is to say, when the reconstructed image is enlarged and outputted for the printing output and the screen display, the above-mentioned noises stand out awfully, and visually bad influences appear extremely. On the contrary, when the reconstructed image is outputted as a scale-down image, the noises do not stand out and the visually bad influence gets small.
If the noise removal processing is performed without considering those effects, the noises cannot be removed sufficiently. Otherwise, unnecessary big blurring occurs.
Therefore, the invention has an object to provide an image processing apparatus that removes the noises efficiently corresponding to the output of the reconstructed image considering the magnifying ratio at the output of the reconstructed image.